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The  Acquisition  of  Expertise 
Lesgold 

1.0  OVERVIEW 

Many  social  roles  require  substantial  training  if  they  are  to  be 
performed  with  facility  and  accuracy.  The  concern  of  this  chapter  is  with 
roles  that  are  complex  enough  to  require  thousands  of  hours  of  practice, 
such  as  the  ability  to  do  arithmetic  well  enough  to  prepare  a  home  budget, 
the  ability  to  diagnose  disease  from  chest  x-ray  pictures,  the  ability  to 
guess  what  the  Soviet  government  will  do  in  the  face  of  crisis,  the  ability 
to  find  efficient  routes  in  a  large  city  as  taxicab  drivers  do,  and  the 
ability  to  read  best-selling  novels  well  enough  to  make  good  cocktail  party 
conversation.  As  can  be  seen  from  these  examples,  some  forms  of  expertise 
are  part  of  common  roles  and  others  are  to  be  found  only  in  specialists. 

This  chapter  considers  psychological  issues  that  span  both  forms. 

The  folk  view  of  expertise  may  be  a  good  starting  point.  I  suppose 
that  the  average  person  thinks  that  expertise  requires  a  combination  of 
practice,  special  knowledge,  lore,  and  innate  ability.  This  chapter  is 
concerned  with  practice  and  with  the  effects  of  existing  knowledge  on  both 
performance  and  further  learning.  It  does  not  consider  genetic  factors  in 
expertise;  they  are  probably  important  in  super-human  extreme  performances, 
but  research  on  instruction  seems  more  useful  than  research  aimed  at  usin?, 
selection  to  avoid  the  need  for  instruction.  Issues  of  strategy  are  not 
covered  either,  although  strategy  is  an  important  aspect  of  expert 
performance.  In  addition  to  practice  and  knowledge  effects,  the  ability  to 
construct  representations  of  problem  situations  seems  to  be  a  major  aspect 
of  expertise,  and  it  is  discussed  near  the  end  of  the  chapter. 
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Consider  two  metaphors  for  expertise.  The  first  is  the  long  distance 
runner.  This  athlete  prepares  for  his/her  performances  largely  by 
practicing.  S/he  runs  every  day,  building  his/her  endurance,  cutting 
his/her  times.  The  effort  is  relatively  non-specific.  Perhaps  world-class 
runners  give  great  attention  to  the  specific  types  of  practice  they  need, 
but  one  can  hardly  compose  a  list  of  the  1000  things  the  runner  must  learn 
in  order  to  be  successful.  In  contrast  is  the  chess  master,  who  seems  to 
know  just  how  to  respond  to  each  of  thousands  of  moves  made  by  his/her 
opponents.  Such  expertise  is  extremely  knowledge-specific.  If  s/he  played 
the  same  game  every  day,  never  exposing  him/herself  to  new  opposition, 
learning  would  proceed  rather  slowly.  A  chess  master  needs  to  know  about 
the  problems  s/he  faces  in  each  of  hundreds  of  situations.  In  the  sections 
that  follow,  I  will  be  concerned  with  both  aspects  of  learning,  knowledge 
which  comes  from  variation  and  knowledge  which  comes  from  repetition.  Then, 
I  will  examine  the  ability  to  build  useful  mental  representations,  a  central 
capability  that  involves  both  these  aspects.  I  conclude  with  a  few 
recommendations  for  the  training  of  cognitive  skills. 


2.0  THE  ROLE  OF  KNOWLEDGE  IN  EXPERTISE 

2.1  Chess 

Comparing  experts  to  novices  allows  us  to  discover  what  changes  take 
place  as  expertise  is  acquired.  Many  forms  of  expertise  were  studied  by 
early  psychologists.  For  example,  Bryan  and  Harter  (1397)  observed  railroad 
telegraphers  at  various  points  in  their  development  of  Morse  code  skills. 
However,  it  is  convenient  to  begin  our  development  with  recent  studies  of 
chess  experts  by  de  Groot  (1966)  and  Chase  and  Simon  (1973a,  1973b).  This 
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work  extended  the  quantitative  expert-novice  comparison  paradigm  by  shifting 
emphasis  to  qualitative  comparisons  as  well.  Some  of  the  results  were 
rather  surprising. 

Prior  to  de  Groot's  (1966)  work,  it  was  generally  thought  that  the 
expert  chess  player  is  the  one  who  can  think  many  moves  ahead  in  the  game, 
following  up  on  the  implications  of  every  possible  move,  while  the  novice  is 
tripped  up  because  s/he  fails  to  think  far  enough  ahead,  something  that 
presumably  comes  with  practice  and  perhaps  Innate  ability.  However,  de 
Groot  found  that  neither  experts  nor  novices  think  ahead  more  than  a  few 
moves;  if  anything,  it  was  the  intermediate  level  player  who  did  more 
thinking  ahead,  following  up  on  the  consequences  of  bad  moves  as  well  as 
good  ones.  Nor  were  there  any  differences  in  the  number  of  moves  considered 
or  in  the  heuristics  used  to  consider  the  consequences  of  those  moves.  What 
experts  could  do  better  was  to  temporarily  remember  board  positions. 

Masters  could  remember  any  real  game  board  after  seeing  it  for  five  seconds, 
although  they  did  no  better  than  weaker  players  in  remembering  random 
arrangements  of  chess  pieces  on  the  board. 

William  Chase  and  Herbert  Simon  were  inspired  by  this  work  to  conduct 
an  extensive  amount  of  experimentation  on  chess  experts  (Chase  &  Simon, 
1973a,  1973b;  Simon  &  Barenfeld,  1969;  Simon  &  Gilmartin,  1973).  The 
outgrowth  of  their  work  was  a  picture  of  the  chess  master  as  a  person  who 
can  recognize  10,000  to  100,000  different  meaningful  board  positions  and 
make  the  optimal  response  to  each.  As  Newell  (1973)  has  pointed  out,  this 
description  suggests  a  theory  of  expertise — that  the  expert  can  recognize 
each  situation  which  s/he  might  encounter  and  has  associated  each  with  a 
specific  response.  Such  a  theory  of  knowledge-specific  expertise  might  seem 
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to  offer  a  rather  dismal  future  for  cognitive  psychologists  studying 
expertise,  in  which  we  become  mere  cataloguers  of  the  thousands  of 
microskills  which  constitute  expertise.  What  has  kept  us  going  is  the 
belief  that  there  are  elements  of  common  structure  in  the  memory  for  these 
patterns  and  in  the  mappings  of  pattern  memory  onto  memory  for  appropriate 
actions.  Nonetheless,  work  on  chess  suggests  that  a  wide  variety  of 
specific  knowledge  must  be  learned  in  order  to  become  an  expert. 

2.2  Reading 

The  general  skill  of  reading  also  depends  on  specific  knowledge.  Voss 
and  his  research  group  have  studied  the  role  of  domain-relevant  knowledge  in 
reading  and  related  literate  performances  (Chiesi,  Spilich,  &  Voss,  1979; 
Spilich,  Vesonder,  Chiesi,  &  Voss,  1979;  Voss,  Vesonder,  &  Spilich,  1980). 
In  some  respects,  the  picture  they  have  built  is  similar  to  that  of  the 
chess  master.  People  who  know  a  lot  are  better  than  the  less  expert  at 
recalling  stories  relevant  to  their  expert  knowledge.  Voss  has  also  found 
that  high-knowledge  people  use  their  knowledge  to  anticipate  what  a  text 
will  say,  as  an  anchor  for  information  that  must  temporarily  be  held  in 
mind,  and  as  a  framework  that  permits  recall  of  more  elaborating  detail. 
These  effects  are  knowledge-specific,  just  as  was  found  in  the  case  of 
chess.  That  is,  the  exceptional  capability  is  to  be  found  only  when  the 
text  deals  with  a  person's  areas  of  expertise. 

The  evidence  for  the  assertions  just  made  comes  from  experiments  in 
which  Voss  and  his  colleagues  tested  a  pool  of  people  for  knowledge  of  the 
intricacies  of  baseball.  They  identified  a  group  of  subjects  (the 
high-knowledge  pool),  who  knew  quite  a  bit  about  the  strategies  of  the 
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garae[l]  and  a  second  group  (the  low-knowledge  pool),  who  knew  the  basic 
rules  of  the  game  and  a  bit  about  which  professional' teams  were  currently 
doing  well  but  little  about  the  game's  finer  points.  These  subjects  were 
asked  to  produce  a  narrative  account  of  a  half  inning  of  a  fictitious 
baseball  game.  Both  groups  produced  about  the  same  amount  of  text  and  had 
about  the  same  types  of  basic  plots.  However,  the  high-knowledge  group 
devoted  more  text  to  elaboration  of  game  activities  such  as  changes  in  the 
basic  state  of  the  game,  while  the  low-knowledge  group  produced  more 
statements  on  irrelevant,  nongame  activities.  The  subjects  were  asked,  two 
weeks  later,  to  recall  what  they  had  written.  In  their  recalls,  the 
low-knowledge  subjects  were  less  likely  to  correctly  recall  what  happened  to 
each  batter  (e.g.,  pop  fly,  single,  strikeout),  were  less  able  to  reproduce 
the  basic  action  sequence,  and  reproduced  less  game-relevant  detail. 

Related  work  has  been  done  which  deals  more  specifically  with  reading. 
For  example,  Splllch  et  al.  (1979)  attempted  to  apply  a  quantitative  model 
of  the  short-term  memory  dynamics  of  reading  to  the  task  of  specifying  how 
the  abilities  of  high-knowledge  people  differ  from  those  of  people  with  less 
knowledge.  Using  a  variant  of  the  Kintsch  and  van  Dijk  (1978)  model  of  text 
comprehension,  they  showed  (a)  that  the  short-term  memory  dynamics  for  the 
main  points  of  a  passage  differed  from  those  for  details,  and  (b)  that 
high-knowledge  individuals  could  hold  more  main-point  (macrostructure) 
information  in  mind  while  reading  a  text  relevant  to  their  expertise. 

Chiesi  et  al.  (1979)  asked  subjects  to  write  down  all  of  the  possible 
outcomes  they  could  think  of  for  a  specific  baseball  situation.  They  found 
that  high-knowledge  individuals  knew  more  possible  outcomes  and  could  better 
specify  which  ones  were  likely  to  occur. 
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This  leads  to  the  following  basic  account  of  the  effects  of  knowledge 
on  expertise.  Experts  can  use  their  knowledge  to  keep  track  of  the 
information  they  are  being  given  when  they  read.  Their  effective  short-term 
memory  is  greater,  especially  for  the  "main  plot"  of  the  material  they  are 
reading.  Their  knowledge  of  the  kinds  of  events  that  can  occur  in  their 
domains  of  expertise  allows  them  to  more  easily  remember  complicated  events. 
Their  ability  to  anticipate  what  is  likely  to  happen  greatly  decreases  the 
effort  they  must  invest  to  understand  a  text.  Thus  far,  we  do  not  have  a 
good  sense  of  the  specifics  of  knowledge  acquisition  effects  on  reading 
acquisition.  That  is,  we  do  not  know  what  kind  of  knowledge  acquiring 
activity  is  optimal  for  improving  reading  skill,  although  considerations  in 
later  sections  of  this  chapter  may  be  relevant. 

2.3  Problem  Solving  In  Less-Structured  Domains 

Knowledge  is  also  an  important  key  to  expertise  in  problem  solving. 
Voss,  Tyler,  and  Yengo  (in  press)  conducted  an  important  study  in  which  they 
asked  political  scientists  specializing  in  the  Soviet  political  system  to 
solve  the  following  problem  and  others  like  it  while  thinking  out  loud: 

Assume  you  are  the  Head  of  the  Soviet  Ministry  of  Agriculture,  and 
assume  crop  productivity  has  been  low  over  the  past  several  years. 

You  now  have  the  responsibility  to  increase  crop  production.  How 
would  you  go  about  doing  this? 

As  a  control  over  the  contributions  of  general  political  science  reasoning 
strategies  and  even  more  general  scientific  reasoning  methods,  Voss  et  al. 
also  used  subjects  who  were  political  scientists  with  other  specialties, 
scientists  in  a  totally  different  field  (chemistry),  and  undergraduate 
olltica*  clence  majors. 


7 


The  Acquisition  of  Expertise 
Lesgoli 

Voss  et  al.  examined  the  extent  to  which  effort  was  invested  in 
building  an  initial  representation  of  the  problem  situation.  For  the. 
problem  stated  above,  this  would  include  the  current  agriculture  scene  in 
the  Soviet  Union,  relevant  political  constraints,  peculiar  aspects  of  Soviet 
agricultural  activity,  and  similar  data.  In  fact,  the  non-specialist 
political  scientists  devoted  more  of  their  thinking  aloud  (16%  of  the  nodes 
in  their  protocol  graphs)  to  this  initial  representation  information  than 
did  the  chemists  (1%)  or  the  students  (0%).  However,  the  specialists 
devoted  the  most  effort  of  all  (24%)  to  initial  representation  of  the 
problem.  This  suggests  that  specific  knowledge  is  an  important  factor  in 
the  creation  of  such  representations  and  that  very  little  expertise  rests  on 
fully  general  strategies  (else  the  chemists  would  have  looked  more  expert). 

Soviet  area  experts  tended  to  offer  a  small  number  of  solutions  that 
were  stated  in  rather  abstract  form,  developed  in  some  depth,  and  backed  by 
detailed  support.  By  depth,  Voss  et  al.  meant  the  size  of  the  chain  of 
causes  that  backed  up  an  assertion.  Expert  arguments  had  three  times  as 
deep  a  structure  of  detailed  support  than  did  those  of  novices.  Also,  much 
more  of  the  expert  analysis  was  an  analysis  of  an  abstracted  representation 
of  the  problem  rather  than  of  the  specifics  of  the  problem  statement. 

This  two-stage  approach  (representation,  then  solution)  usually  pays 
off  by  decreasing  the  amount  of  thinking  needed  to  produce  a  good  solution. 

A  good  abstracted  problem  representation  will  adequately  capture  the 
constraints  that  are  relevant  to  a  solution,  while  working  directly  from  the 
specific  problem  statements  leaves  the  resolution  of  these  constraints  as  a 
separate  task  to  be  done  after  a  potential  unwieldy  set  of  possible 
solutions  is  generated.  One  example  that  arose  in  Voss's  laboratory 
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illustrates  this  problem.  An  expert,  at  the  beginning  of  his  analysis  on 
the  agriculture  problem,  pointed  out  that  the  problem  involved  several 
ministries  and  the  entire  agriculture  system,  from  its  raw  materials  of 
fertilizer,  seed,  labor,  etc.,  to  its  ability  to  distribute  the  final 
product.  By  thinking  this  way,  he  was  able  to  keep  in  mind  from  the  outset 
that  any  changes  requiring  more  fertilizer  might  fail  because  substantial 
amounts  of  Soviet  fertilizer  are  lost  due  to  inadequate  packaging. 
Information  about  the  ministry  that  controlled  bag  making  needed  to  be 
considered  from  the  outset. 

Voss,  Greene,  Post,  and  Penner  (in  press)  have  suggested  some  of  the 
changes  that  take  place  in  the  course  of  acquiring  expertise  in  a  domain, 
such  as  political  science,  in  which  problems  are  wide  ranging  and  solutions 
not  easily  verified.  They  noticed  that  graduate  students  in  this  area 
showed  three  differences  from  undergraduates  in  their  protocols:  they  had 
some  knowled-'.  of  subproblera  interactions  (ways  in  which  solving  one 
subproblem  might  interfere  with  a  solution  that  otherwise  makes  sense  for 
another),  their  descriptions  of  the  problem  situation  were  more  abstract, 
and  their  reasoning  in  support  and  evaluation  of  their  plans  more  extensive. 
From  these  differences,  Voss  et  al.  concluded  that  the  graduate  students 
have  more  complete  knowledge  networks,  containing  more  explicit  causal 
knowledge  and  organized  more  hierarchically. 

Experts,  they  concluded,  presented  further  knowledge  development  as 
well  as  refined  discipline-specific  and  domain-specific  strategies  for  using 
that  knowledge.  Some  of  their  strategies  were  general  across  all  of 
political  science,  while  others,  such  as  extensive  use  of  historical 
analysis,  were  more  specific  to  the  Soviet  situation.  Finally,  et  al. 
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noted  the  importance  of  expert  flexibility  in  using  knowledge  and  suggested 
that  this  flexibility  comes  from  experience  with  a  wide  variety  of  problems. 
In  certain  other  domains ,  such  as  engineering  or  physics ,  where  problem 
solutions  are  more  clearly  defined,  the  number  of  experiences  with  problems 
of  a  single  type  might  be  more  important;  in  political  science,  experience 
with  problem  variety  seemed  of  especial  centrality. 

2.4  Vocabulary 

The  work  in  Voss's  laboratory  has  concentrated  on  the  effects  of 
knowing  a  lot  about  a  specific  subject  on  understanding  discourses  or 
solving  problems  involving  that  subject.  It  is  also  possible  to  consider 
the  contribution  of  knowledge  at  a  less  specific  level,  namely  general 
vocabulary.  Virtually  every  intelligence  or  verbal  aptitude  test  includes  a 
vocabulary  component  because  vocabulary  is  so  predictive  of  verbal 
competence.  My  colleagues  Beck,  Perfetti,  and  McKeown  (in  press)  and  Glaser 
and  Curtis  (personal  communication)  have  been  trying  to  discover  why 
extensive  and  speedy  word  knowledge  is  important  in  the  overall  acquisition 
of  reading  skill  and  other  verbal  competences.  Since  vocabulary  is  often 
tested  in  aptitude  tests,  Glaser  and  Curtis  undertook  the  task  of 
determining  the  specific  vocabulary  capabilities  that  distinguish  the  high 
verbal  person  from  the  less  skilled.  Their  basic  findings  have  been  that 
the  high  verbal  person  not  only  knows  the  definitions  of  more  words,  but 
also  has  more  knowledge  that  relates  to  each  known  word.  Thinking  of  human 
knowledge  as  a  network  of  conceptual  relationships,  we  can  describe  their 
results  as  showing  that  the  high  verbal  person  has  more  words  tied  into 
his/her  network  and  also  has  more  links,  on  average,  from  any  given  word's 
encoding  to  other  concepts  that  s/he  knows  about.  Much  of  our  power  to 
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understand  complex  or  ambiguous  text  rests  in  word-specific  knowledge  (e.g., 
Small,  1980). 

Classroom  research  by  Beck  et  al.  (in  press)  strongly  suggests  that 
reading  ability  can  be  improved  by  a  vocabulary  training  program  that 
emphasizes  the  speed  of  access  for  word  knowledge  as  well  as  the  richness  of 
that  knowledge.  Fourth  grade  children  were  taught  approximately  LOO  words 
over  a  five-month  period.  Following  instruction,  the  children  performed 
tasks  designed  to  require  semantic  processes  ranging  from  single  word 
semantic  decisions  to  simple  sentence  verification  and  memory  for  connected 
text.  On  all  these  tasks,  instructed  subjects  performed  at  a  significantly 
higher  level  than  control  subjects  matched  on  measures  of  vocabulary  and 
comprehension  prior  to  Instruction.  Further,  words  for  which  more 
instruction  was  provided  were  processed  more  quickly  by  the  subjects  than 
words  for  which  they  had  received  less  special  intervention.  Instructed 
subjects  learned  the  word  meanings  taught  by  the  program  and  used  instructed 
words  more  efficiently  in  tasks  involving  comprehension.  Indeed,  they  even 
improved  on  standardized  reading  comprehension  tests  faster  than  their 
matched  control  classmates. 

This  work  leaves  unresolved  the  relative  importance  of  breadth  of 
vocabulary  knowledge  and  level  of  efficiency  or  facility  that  is  needed. 

One  possibility  consistent  with  these  results  is  that  knowing  all  sorts  of 
things  about  the  words  one  is  likely  to  encounter  is  the  key  to  successful 
reading.  On  the  other  hand,  the  critical  factor  may  be  the  extent  to  which 
the  words  one  does  know  have  been  practiced  sufficiently  to  produce 
capabilities  that  can  execute  without  substantial  conscious  planning.  Quite 
probably,  both  of  these  effects  are  involved  in  reading;  there  were  some 
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indications  of  this  in  the  Voss  work,  on  political  reasoning  described  above. 
Issues  of  practice  and  process  automation  will  be  considered  later  in  this 
chapter. 

2.5  Physics 

Quite  a  bit  of  the  work  on  differences  between  novices  and  experts  has 
been  done  in  the  domain  of  physics  (e.g.,  Chi,  Glaser,  &  Rees,  in  press; 
Larkin,  McDermott,  Simon,  &  Simon,  1981;  McCloskey,  Cararaazza,  &  Green, 
1980).  A  central  finding,  stated  more  or  less  explicitly  in  different 
studies,  is  that  representing  the  problem  is  a  central  part  of  problem 
solving  for  experts.  Novices  tend  to  invoke  equations  quickly,  selecting 
those  that  include  both  what  is  given  and  what  is  to  be  found.  In  contrast, 
experts  concentrate  first  on  understanding  the  problem.  Experts  are  more 
likely  to  draw  diagrams,  for  example,  and  a  bigger  proportion  of  their  total 
problem  solving  time  elapses  before  they  write  any  equations.  Once  the 
representation  is  built,  the  solution  methods  also  differ.  Simon  and  Simon 
(1978)  found  that  novices  are  more  likely  to  use  a  working  backwards 
strategy  when  solving  physics  problems,  while  experts  are  more  likely  to  use 
a  working  forward  strategy. 

One  might  initially  be  tempted  to  say  that  the  novice  merely  needs  to 
be  taught  the  working  forward  strategy.  This  is  not  likely  to  be 
sufficient.  The  problem  is  that  a  working  forward  strategy  requires  (a)  a 
sufficient  representation  of  the  problem  from  which  straightforward 
inferences  can  lead  to  solution  and  (b)  enough  knowledge  of  the  course  such 
inference  will  take  to  permit  focusing  attention  on  those  partial  solutions 
that  are  most  promising.  A  working  backwards  strategy,  in  contrast, 
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requires  little  goal-related  knowledge.  At  each  step  along  the  way,  there 
is  a  list  of  things  which,  if  known,  would  result  in  solution.  Less 
specific  knowledge  is  required  to  use  such  a  strategy  but  more  partial 
results  oust  be  kept  in  short-term  memory. 

The  importance  of  knowledge  is  also  illustrated  by  repeated  findings 
that  people,  even  those  who  have  taken  a  term  of  physics,  maintain  very 
naive  views  about  the  effects  of  forces  on  objects  (McCloskey  et  al.,  1980). 
Even  after  they  have  spent  a  term  solving  mechanics  problems,  students' 
beliefs  about  the  world  are  inconsistent  with  Newton's  laws.  For  example, 
they  believe  that  a  body  in  motion  will  change  speed  even  when  no  force  is 
applied.  Their  qualitative  predictions  about  physical  events  are  often 
incorrect  even  though  they  are  able  to  solve  equations  and  quantitative 
problems  about  force,  mass,  acceleration,  velocity,  and  displacement 
correctly.  When  we  look  at  how  physics  is  taught,  we  get  some  sense  of  why 
this  might  be  so. 

Much  of  an  introductory  physics  course  consists  of  solving  numerical 
problems  about  masses,  movements,  and  energy.  Thus,  there  is  emphasis  on 
the  fundamental  relationships,  as  captured  by  equations.  Sufficient 
richness  and  facility  in  mentally  representing  physical  events  takes  longer 
to  develop  and  involves  not  only  knowing  the  principles  in  declarative  form 
but  also  developing  procedures  for  mapping  those  principles  onto  concrete 
situations.  There  are  no  final  exams  in  mental  representing,  while  there 
are  tests  requiring  the  knowledge  of  formulas  and  the  use  of  such  formulas 
to  solve  problems. 


13 


The  Acquisition  of  Expertise 
Lesgold 

2.6  Radiological  Diagnosis 

To  further  illustrate  the  importance  of  organized  knowledge,  I  next 
discuss  some  work  that  Robert  Glaser,  Paul  Feltovich,  Yen  Wang,  Harriet 
Rubinson  and  I  have  been  doing  on  radiological  expertise  (Lesgold, 

Feltovich,  Glaser,  and  Wang,  1981,  describes  the  earlier  parts  of  this 
work).  Part  of  that  work  deals  with  the  way  in  which  physicians  acquire  the 
organized  bodies  of  knowledge  (schemas)  that  constitute  radiological 
anatomy,  the  science  of  relationships  between  anatomical  structure 
variations  and  x-ray  plate  patterns.  To  provide  a  richer  sense  of  the 
development  of  expert  knowledge  and  to  give  a  sense  of  the  data  with  which 
one  deals  in  this  area,  this  work  is  presented  in  some  detail.  This  is 
necessary  in  order  to  provide  a  sense  of  the  dependence  of  the  meaning  of 
individual  perceptual  features  on  a  complex  decision-making  context. 

When  a  radiology  resident  in  a  teaching  hospital  learns  a  complex 
diagnostic  schema  for  a  disease,  s/he  is  initially  in  a  very  precarious 
position.  This  precariousness  is  illustrated  by  data  from  subjects  who  were 
asked  to  diagnose  a  film  showing  a  collapsed  lung  lobe  (atelectasis).  The 
most  obvious  cue  for  collapsed  lung  is  the  presence  of  a  local  increase  in 
tissue  density  (a  white  region  on  the  film;  cf.  Figures  1  and  2).  Certain 
properties  of  the  density  (e.g.,  "triangular,"  "sharp  borders,"  etc.)  make 
it  more  likely  to  reflect  atelectasis,  but  even  a  density  with  those  special 
features  is  insufficient  for  a  certain  diagnosis  of  atelectasis,  since  other 
diseases  could  produce  the  same  sign.  Listed  below  are  six  other  general 
indicators  for  atelectasis;  these  Include  indirect  signs  of  the  changes 
that  occur  throughout  the  chest  when  the  space  taken  up  by  a 
previously-inflated  lung  lobe  becomes  unoccupied. 
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o  Mediastinal  shift :  the  anatomy  contained  between  the  lungs  is 
pushed  to  one  side  or  the  other; 

o  Displaced  fissure:  the  boundaries  between  lung  lobes  are  not  in 
their  normal  locations; 

o  Elevation  of  the  diaphragm:  the  diaphragm  is  pressed  higher  into 
the  chest  than  is  usual; 

o  Harrowing  of  the  rib  cage; 

o  Compensatory  hyperinflation:  the  noncollapsed  lung  balloons 
because  of  decreased  external  pressure; 

o  Hilar  displacement:  the  major  vascular  structures  feeding  the 
lungs  are  not  in  their  normal  places. 

He  assume  that  the  atelectasis  schema  is  triggered  by  the  presence  of  a 
subset  of  the  seven  features  and  that  all  seven  are  conditional  criteria  for 
supporting  the  diagnosis  over  alternatives.  However,  the  indirect  features 
vary  with  the  location,  severity,  and  recency  of  the  collapse.  A 
radiologist  must  have  a  well-tuned  understanding  of  dynamic  changes  which 
occur  when  the  lung  loses  volume  and  a  mapping  of  that  understanding  onto 
procedures  for  constructing  the  representation  of  a  chest  from  a  film.  The 
presence  of  one  feature  may  increase  the  criterlal  importance  of  another, 
while  decreasing  the  expectation  for  a  third.  Verifying  atelectasis 
requires  a  mental  representation  of  the  patient  that  takes  account  of  these 
complexities . 
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The  predicament  of  the  novice  can  be  illustrated  with  the  following 
simplified  account  of  a  series  of  simulations  that  we  conducted.  In  these 
simulations,  there  were  rules  which  represent  the  triggering  conditions  for 
schemata.  For  example,  a  general  rule  for  atelectasis  states  that 


Rule  U 

IF  you  have  seen  mediastinal-shift,  THEN 
ASSUME  a  diagnosis  of  atelectasis 
IF  ANY  OF  THE  FOLLOWING  have  been  seen: 
displace-fissure 
elevated-diaphragm 
narrowing-rib-cage 
compensatory-hyperinflation 
hilar-displacement 
local-increase-in-density. 


Going  further,  it  seems  likely  that  once  such  a  rule  has  triggered  the 
atelectasis  schema,  one  would  expect  to  see  the  other  features.  The  next 
rule  provides  for  this  conditional  expectation. 


Rule  2^: 

IF  the  diagnosis  is  atelectasis,  THEN  expect  to  see 
ALL  of  the  following: 
hilar-displacement 
compensatory-hyperinflation 
displace-fissure 
elevated-diaphragm 
narrowing-rib-cage 
mediastinal-shift 
local-increase-in-density. 


Finally,  it  seems  reasonable  to  raise  a  flag  whenever  one  cannot  see 
something  one  expected  to  see,  as  shown  in  the  next  example: 
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Rule  3: 

IF  you  expect  to  see  X  BUT  you  don't,  THEN 
DECLARE  A  CONTRADICTION. 


This,  though,  sets  the  stage  for  a  situation  in  which  ordinary  truth 
maintenance  operations [2J  will  cause  the  physician  to  talk  himself  out  of 
atelectasis  after  first  talking  himself  into  it,  at  least  in  some  cases. 
Consider  the  case  in  which  there  is  mediastinal  shift  and  hilar  displacement 
but  no  visible  displacement  of  the  appropriate  fissure.  For  some  of  our 
subjects,  this  is  what  they  saw  on  our  film. 

They,  and  our  simulation,  reacted  to  the  contradiction  between  the 
inference  of  atelectasis  and  the  failure  to  see  the  displaced  fissure  by 
dismissing  the  atelectasis  hypothesis.  One  of  our  subjects,  a  second-year 
resident,  said  this  very  directly: 

"I  am  also  indicating,  drawing  a  line,  over  what  I  think  is 
the  minor  fissure  because  the  other  differential  to  be 
considered  is  right  middle  lobe  collapse,  and  there's  no 
depression  of  the  minor  fissure  or  loss  of  volume  in  the 
right  lung  lobe  to  support  right  middle  lobe  collapse.  I'm 
also  showing  that  the  right  hemidiaphragra  is  in  proper 
position  and  not  elevated  as  you  would  see  in  right  middle 
lobe  collapse.” 

This  subject  needed  each  of  several  signs  to  be  present  to  be  sure  of  his 
diagnosis.  He  missed  some  that  were  present  but  more  subtle  and  attenuated 
than  usual.  Others  sometimes  do  not  appear  in  this  specific  kind  of  case. 

There  is  some  complexity  to  what  a  successful  subject  should  have  done 
on  the  film  in  question.  An  expert  would  have  known  that  when  the  specific 
kind  of  atelectasis  shown  in  Figure  1  occurs,  the  fissure  is  hard  to  see  in 
a  front  view,  his/her  recognition  capability  would  be  tuned  to  expect  the 
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fissure  to  be  hidden  in  certain  cases.  Me  can  represent  this  by  changing 
the  rule  that  sets'  our  expectations  for  atelectasis  to  include  the  fact  that 
we  really  only  expect  a  fissure  displacement  to  show  in  a  lateral  view. 

Some  of  our  more  expert  subjects  have  elaborated  their  knowledge  in  a 
somewhat  different  way.  Specifically,  they  have  learned  special  indications 
for  different  atelectasis  forms.  There  is  a  specific  feature  in  our  film 
which  strongly  signals  the  possibility  of  atelectasis,  even  in  the  absence 
of  some  of  the  usual  signs.  A  sharply  pointed,  sail-shaped  density  is 
located  near  the  right  heart  border.  This  is  the  shadow  of  the  deflated 
right  middle  lobe  itself,  which  is  sandwiched  between  the  upper  and  lower 
lobes. [4]  The  location  and  unique  characteristics  of  the  density  are  an 
important  additional  cue,  which  should  be  incorporated  into  a  specialized 
schema  for  right  middle  lobe  collapse. 

In  both  our  simulations  and  our  data,  the  diagnosis  of  atelectasis 
raised  by  the  general  atelectasis  schema  is  rejected  because  of  a 
contradiction  of  expectation  with  observation,  as  in  our  first  simulation, 
but  the  diagnosis  of  right-middle-lobe  chronic  atelectasis  made  by  a  more 
specific  schema-triggering  rule  is  retained. 

What  can  we  learn  from  this  type  of  data  analysis  and  modeling? 
Basically,  two  points  are  made  more  apparent.  First,  an  important  way  in 
which  knowledge  is  organized  is  with  schemata.  Schemata,  for  our  purposes, 
are  sets  of  assumptions  and  rules  for  interpreting  new  information  that  are 
triggered  when  certain  conditions  are  satisfied.  A  likely  trigger  mechanism 
is  the  presence  of  information  that  confirms  a  threshold  number  of  the 
schema's  assumptions,  but  more  refined  trigger  mechanisms  probably  are 
learned  with  sufficient  practice. 
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The  second  conclusion  drawn  from  our  results  is  that  there  have  to  be 
ways  in  which  a  schema  can  spawn  an  offspring  that  is  a  more  specific  and 
detailed  version  of  the  original.  In  this  "clone-and-ref ine"  process,  the 
initial  atelectasis  schema  is  retained,  but  a  specialized  schema  also 
develops,  which  is  a  particularized  expansion  of  the  initial  one.  This 
specialized  schema  jLs  not  merely  a  reweighting  of  the  diagnosticity  of 
different  perceptual  features  but  reflects  weightings  of  some  features  that 
are  contingent  on  which  other  features  have  been  noticed.  These  specialized 
schemata  may  operate  at  the  perceptual  level  in  domains  like  radiology.  In 
a  domain  like  chess,  they  may  operate  at  a  slightly  higher  level.  A  rook  in 
a  particular  space  will  not  be  misinterpreted  as  a  pawn;  the  contingent 
meaning  lies  in  how  the  locations  of  other  pieces  and  the  movement 
restrictions  they  have  bias  the  meaning  of  the  rook's  location. 

Much  work  remains  to  be  done  in  order  for  a  detailed  model  of  the 
schema  specialization  process  to  be  specified  and  tested.  Two  principles 
might  guide  further  theorizing.  First,  while  many  aspects  of  expertise  are 
best  understood  when  studied  at  the  level  of  schemata,  some  aspe:ts  of 
learning  may  be  easier  to  understand  at  a  more  microscopic  level.  Second, 
learning  must,  in  some  way,  operate  on  a  trace  of  the  recent  course  of 
thinking.  That  is,  the  episodic  memory  trace  of  recent  schema  activity, 
recent  actions  (mental  or  physical),  and  the  immediate  consequences  of  those 
actions  must  be  the  foundation  for  new  learning.  I  next  consider  a  number 
of  learning  mechanisms  that  are  consistent  with  these  principles. 
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2.7  Learning  And  Refining  Cognitive  Skills 

John  Anderson  (1982)  has  recently  developed  a  theory  of  skill 
acquisition  that  is  a  good  starting  point  for  understanding  the  initial 
acquisition  of  specific  knowledge  and  the  effects  of  practice.  Taking  the 
work  of  Fitts  (1964)  as  a  starting  point,  the  theory  divides  the  course  of 
learning  into  three  parts:  the  declarative  stage,  the  knowledge  compilation 
stage,  and  the  procedural  stage.  Anderson's  theory  allows  a  clearer 
understanding  of  both  the  types  of  knowledge  that  are  involved  in  skill  and 
how  each  of  those  types  is  acquired.  I  shall  sketch  its  main  points  (see 
Anderson,  1982,  for  details). 

Anderson  proposes  that  initial  performance  in  a  novel  situation 
involves  the  operation  of  general  strategies  that  use  declarative  knowledge 
to  guide  performance.  In  our  case,  for  example,  we  can  imagine  some  very 
general  strategies  for  diagnosis  that  a  physician  might  acquire  in  medical 
school.  The  basic  process  might  go  something  like  this: 

a.  Make  a  list  of  all  abnormalities  or  patient  complaints  you  can 
notice. 

b.  Is  there  a  disease  that  you  know  to  be  associated  with  all  of  these 
complaints?  If  there  is  exactly  one,  then  it  is  the  appropriate 
diagnosis,  so  stop.  If  there  are  more  than  one,  then  go  to  £.  If 
no  disease  matches  perfectly,  go  to  d_. 

c.  Search  memory  for  information  on  data  that  might  separate  the 
candidate  uiseases.  Collect  some  of  chat  data  and  recur  through 
the  procedure. 


21 


The  Acquisition  of  Expertise  . 

Lesgold 

d.  Make  a  list  of  the  diseases  that  are  consistent  with  the  symptoms. 
Look  for  data  which  is  needed  to  confirm  one  or  more  of  the 
candidates  on  the  list  and  recur  through  this  process.  If  you  hit 
a  dead  end,  then  use  other  procedures  to  decide  which  of  the 
potential  candidates  to  pick  (use  of  other  procedures  might  depend 
upon  the  cost  of  treatment  and  the  consequences  of  nontreatment). 

By  reading  textbooks,  receiving  advice  from  attending  physicians,  and 
monitoring  one's  success  in  patient  management,  a  variety  of  bits  of 
declarative  knowledge  (facts  about  how  to  perform  rather  than  procedures  for 
performance)  can  be  acquired.  However,  these  bits  of  knowledge  must  be 
accessed  as  needed  by  the  general  diagnosis  procedure,  a  slow  process 
subject  to  capacity  limits.  Much  of  the  training  time  that  is  invested  by 
radiologists  involves  the  conversion  of  slow  declarative  knowledge 
interpretation  into  faster  compiled  procedures. 

Anderson  suggests  that  we  think  of  the  second  stage,  knowledge 
compilation  as  being  somewhat  analogous  to  compilation  of  a  computer 
program.  Languages  such  as  BASIC  and  LISP  permit  programs  to  be  specified 
in  two  forms,  interpreted  and  compiled.  Compiled  knowledge,  like  compiled 
computer  programs,  runs  faster,  but  at  the  cost  of  greater  difficulty  of 
modification.  Actually,  the  situation  is  a  bit  more  complex,  according  to 
Anderson.  Newly  acquired  knowledge  should  not  be  trusted  as  much  as 
well-practiced  procedures.  Hence,  it  is  useful  for  it  to  have  effect  only 
as  a  result  of  conscious  processing.  Compiled  procedures,  according  to 
Anderson,  are  relatively  automatic.  They  are  in  the  form  of  productions,  or 
condition-action  pairs.  When  a  production's  conditions  are  satisfied,  it 
will  act  (within  the  constraints  of  an  execution  discipline  that  may  limit 
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how  many  productions  can  act  at  once).  The  only  conscious  control  over  a 
production  tends  to  lie  in  the  fact  that  some  of  the  conditions  for 
productions  are  likely  to  be  goal  states  that  can  be  consciously  set. 
Nonetheless,  a  production  may  well  fire  accidentally  even  if  the  goal  states 
do  not  match  perfectly  (Norman,  1981).  Consequently,  there  is  probably 
adaptive  significance  to  the  Anderson  formulation,  in  which  new  knowledge  is 
slowly,  but  consciously,  processed. 

Anderson  proposes  two  processes  for  knowledge  compilation.  First, 
there  is  proceduralization,  in  which  snapshots  are  made  of  a  successful, 
just-completed  activity.  The  scenario  proceeds  as  follows: 

o  A  compiled  general  strategy  procedure  finds  a  potentially  useful 
piece  of  declarative  knowledge  and  sets  up  a  task  of  checking 
conditions  of  the  knowledge  and  then  performing  the  relevant 
actions . 

o  The  task  is  successful  in  achieving  a  conscious  goal. 

o  A  snapshot  of  the  conditions  at  the  time  of  the  successful  action 
is  combined  with  a  snapshot  of  the  action  itself  and  is  stored  in 
memory  as  a  production. 

Anderson's  second  compilation  process  is  composition.  Composition  is 
the  process  whereby  two  productions  which  execute  successfully  in  immediate 
sequence  can  be  combined  into  a  single  production.  The  conditions  for  the 
composed  production  are  the  conditions  for  the  original  first  production 
plus  those  conditions  of  the  second  that  are  not  created  by  the  first.  The 
action  is  the  sequence  of  the  two  original  actions.  This,  then,  is 
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essentially  an  abbreviation  process,  while  proceduralization  is  an 
automation  process. 

In  Anderson's  third  stage,  the  newly  acquired  productions  are  tuned. 

The  snapshot  process,  based  upon  recording  specific  successful  applications 
of  declarative  knowledge,  is  likely  to  result  in  productions  that  are  too 
specific.  Also,  there  are  bound  to  be  occasions  when  a  piece  of  declarative 
knowledge  is  successful  for  accidental  reasons.  For  example,  if  you  had  a 
production  that  diagnoses  atelectasis  every  time  you  see  a  sail-shaped 
density,  you  would  have  a  skill  that  is  correct  part  of  the  time  but  not 
enough  to  be  useful,  since  some  other  diseases  you  had  not  been  told  about 
can  mimic  the  conditions  to  which  you  were  sensitive. 

Tuning  is  accomplished  by  three  mechanisms  that  look  a  lot  like  those 
of  behavioral  learning  theory  except  that  they  Jte  on  Wtfeis  between 
mental  events  and  mental  actions  rather  than  between  physical  events  and 
behaviors.  The  first  mechanism,  strengthening,  derives  from  the  mechanism 
that  decides  which  productions  with  satisfied  conditions  will  be  allowed  to 
act.  The  probability  of  selection  is  a  function  of  strength  values  which 
are  assigned  to  each  production.  New  productions  start  out  weak  and  thus 
tend  to  be  executed  mainly  when  their  conditions  match  the  current  mental 
state  more  closely  than  other  productions.  Each  time  they  are  followed  by 
success,  they  become  stronger. 

The  other  two  methods  of  tuning  are  generalization  and  discrimination. 
Generalization  applies  to  two  productions  whose  conditions  have  Identical 
structure  but  slightly  different  content.  The  result  of  genera’ ’ zation  is 
to  build  a  production  that  has  the  same  structure  but  with  a  variable  at  the 
point  of  difference.  For  example,  Productions  1  and  2  can  lead  to  the 
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generalization  shown  as  Production  3. 

[1]  IF  I  need  my  teddy  bear  to  fall  asleep  and  I  am  sleepy,  THEN  look 
for  my  teddy  bear. 

[2]  IF  I  need  my  blanket  to  fall  asleep  and  I  am  sleepy,  THEN  look  for 
my  blanket. 

[3]  IF  I  need  my  X  to  fall  asleep  and  I  am  sleepy,  THEN  look  for  my  X. 

3.0  PRACTICE 

Anderson's  work  helps  to  integrate  knowledge  acquisition  and  practice, 
two  central  components  of  learning.  I  considered  knowledge  acquisition  in 
the  previous  section.  This  section  considers  issues  of  practice,  attending 
to  classical  studies  of  the  effects  of  overlearning  trials  in  simple 
learning  paradigms  and  to  the  effects  of  cognitive  practice  which  might  be 
accounted  for  by  such  theories  as  Anderson's.  Me  start  by  considering 
relevant  work  on  the  effects  of  practice  in  simple  perceptual  search  tasks. 

3.1  Automatic  And  Controlled  Processes 

Schneider  and  Shiffrin  (1977;  Shiffrin  &  Schneider,  1977)  proposed  a 
theory  of  information  processing  that  dealt  with  practice  in  which  the 
mappings  of  inputs  onto  necessary  responses  is  consistent  over  the  course  of 
the  practice.  Such  consistent  practice,  they  asserted,  leads  to  the 
development  of  automated  processing  capability,  in  which  the  input  triggers 
a  response  sequence  that  operates  Independently  of  the  subject's  control, 
requiring  no  attention  or  conscious  processing  while  still  attracting  an 
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investment  of  such  resources.  In  contrast,  responses  that  are  not  yet 
adequately  practiced  or  that  are  not  oonsistently  mapped  onto  possible 
inputs  (which  they  called  controlled  processes)  require  attention,  use 
limited  short-term  capacity,  and  tend  to  be  more  serial  in  nature.  One  can 
think  of  the  distinction  as  being  similar  to  Anderson's  distinction  between 
declarative  and  proceduralized  knowledge. 

Schneider  and  Shiffrin  supported  their  theory  with  data  from 
experiments  in  which  subjects  had  to  search  a  tachistoscopically  presented 
display  for  target  letters.  In  some  conditions,  the  targets  were  from  a  set 
that  were  always  targets.  In  other  cases,  the  same  symbol  could  be  a  target 
on  one  trial  and  a  distractor  on  a  later  trial.  Consistency  of  mapping  led 
to  better  search  performance.  Both  the  number  of  items  in  the  memory  set 
(the  symbols  being  searched  for)  and  the  number  of  items  in  the  display  were 
varied.  Ordinarily,  as  the  size  of  the  memory  set  (e.g.,  Sternberg,  1975) 
or  the  display  set  (Atkinson,  Holmgren,  &  Juola,  1969)  is  increased,  a 
linear  increase  in  response  time  is  observed.  However,  after  consistent 
practice,  the  functions  became  almost  flat,  suggesting  that  processing  was 
not  only  rapid  but  also  parallel  rather  than  serial.  That  is,  subjects' 
response  times  were  consistent  with  a  model  in  which  each  target  letter  had 
a  production  "watching  for  it."  For  inconsistent  mappings,  where  items  were 
targets  on  some  trials  and  distractors  on  others,  this  did  not  happen,  even 
after  substantial  practice.  These  results  support  Anderson's  theory  and 
suggest  the  importance  of  defining  the  meaningful  units  of  a  task  in  ways 
that  preserve  consistent  mappings  over  the  course  of  practice. 
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3.2  Training  Effects  On  Negative  Transfer 

While  a  consistent  relationship  between  mental  events  and  the  mental 
acts  these  events  should  trigger  can  help  induce  efficient  learning,  it  is 
not  always  possible.  It  is  of  great  adaptive  importance  that  a  person  be 
able  to  learn  new  responses  to  the  same  apparent  conditions.  Fortunately, 
there  is  some  indication  that  the  interference  produced  by  inconsistencies 
can  be  overcome  with  sufficient  practice.  Muensterberg  (1889,  as  cited  in 
Siipola  &  Israel,  1933)  claimed  to  have  conducted  relevant  experiments  on 
himself.  Every  so  often,  he  would  switch  his  watch  from  his  right  pocket  to 
his  left  or  back  again.  Each  switch  meant  that  habits  of  reaching  for  his 
watch  on  one  side  would  have  to  be  changed.  He  claimed  that  after 
maintaining  this  inconsistent  practice  regimen  for  long  enough,  the 
temporary  interference  effect  of  each  switch  attenuated  and  eventually 
disappeared. 

This  assertion  was  directly  tested  by  Siipola  and  Israel  (1933)  in  an 
experiment  on  telegraphy.  After  varying  amounts  of  practice  (up  to  an 
average  of  308  trials  for  the  most-practiced  condition),  the  code  was 
changed  so  that  each  code  used  before  was  now  the  code  for  a  different 
letter,  clearly  an  interference  condition.  The  new  telegraphy  scheme  was 
then  practiced  to  a  very  high  level.  The  results  were  expressed  in  terms  of 
percentage  of  positive  or  negative  transfer,  defined  by  (F1-F2)/F1  where  FI 
is  the  amount  of  training  time  needed  to  reach  a  given  performance  criterion 
on  the  first  task  and  F2  is  the  time  needed  to  reach  the  same  criterion  on 
the  second.  The  general  pattern,  in  line  with  earlier  results,  was  that  as 
the  level  of  learning  of  the  first  task  increased,  negative  transfer 
increased  up  to  a  point,  after  which  the  transfer  effects  became  more 
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positive  as  the  level  of  first-task  training  rose  further.  Substantial 
positive  transfer  was  found  if  the  first-task  training  lasted  long  enough. 
The  relationship  between  transfer  and  amount  of  original  training,  then,  was 
U-shaped.  However,  as  the  criterion  for  learning  was  increased,  the 
negative  transfer  effect  extended  to  greater  amounts  of  initial  training, 
suggesting  that  negative  transfer  is  an  increasing  problem  as  the  target 
level  of  ultimate  skill  is  increased. 

How  can  these  findings  be  explained?  Handler  (1954)  examined  the 
U-shaped  functions  that  Siipola  and  Israel  (1933)  and  others  had  reported 
and  theorized  that  they  arose  because  of  a  combination  of  response  learning 
and  association  learning  at  both  the  response  level  and  at  a  cognitive 
level.  Several  aspects  of  Morse  code  task  performance  should  transfer 
positively  when  the  codes  are  reassigned.  First,  the  motor  program  to  send 
each  code  will  still  be  used.  Second,  each  letter  of  the  alphabet  will  have 
to  be  quickly  recognized  in  the  experimental  context.  Third,  a  cognitive 
representation  of  each  code  may  have  to  be  formed.  The  negative  factors 
will  involve  the  ties  between  letter  and  code,  both  direct  and  symbolic. 
Presumably,  the  positive  effects  of  practice  on  the  code  sequences 
eventually  outweigh  the  negative  effects  of  incorrect  pairings. 

A  high  final-task  criterion  may  require  learning  to  the  procedural 
level.  Here,  the  interfering  effects  of  proceduralized  interference  will 
play  a  role.  In  contrast,  a  low  criterion  for  final  task  performance 
primarily  will  involve  use  of  declarative  knowledge  about  the  letter-to-code 
mapping.  Automation  of  the  code-sending  response  will  facilitate  final 
performance  even  if  competing  declarative  knowledge  from  the  initial 
learning  poses  some  problems.  In  the  high  criterion  case,  interference  will 
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be  present  at  both  the  declarative  and  procedural  levels,  making  it  unlikely 
that  automation  of  the  code-sending  response  alone  will  overcome  the 
negative  factors. 

3.3  Extra  Practice  Improves  Speed  Of  Response 

There  are  several  other  types  of  findings  that  help  in  clarifying  the 
role  of  practice.  One  implication  of  multi-stage  skill  acquisition  theories 
such  as  Anderson's  is  that  one  can  expect  to  find  nonlinearities  in  curves 
that  map  measures  of  performance  onto  amount  of  training.  This  is  because 
different  processes  are  involved  at  each  stage.  One  such  finding  is  due  to 
Judd  and  Glaser  (1969).  They  examined  accuracy  and  latency  in  a 
paired-associate  learning  task  and  found  that  for  any  individual  item, 
response  latency  was  constant  until  the  trial  of  last  error.  After  that, 
additional  trials  produced  drops  in  response  time.  Presumably,  the  trial  of 
last  error  is  a  reasonable  indicator  of  a  point  at  or  before  the  point  at 
which  proceduralized  performance  becomes  dominant  over  performance  driven  by 
declarative  knowledge. 

Different  performance  measures,  then,  are  sensitive  to  different  stages 
of  learning.  Also,  it  seems  likely  that  complex  tasks  Involve 
multidirectional  flows  of  control  between  procedural  and 

declaratively-driven  components.  Because  of  the  cognitive  processing  limits 
faced  by  processes  that  use  declarative  knowledge  interpretlvely,  it  should 
often  be  the  case  that  some  of  the  declarative  learning  cannot  take  place 
until  other  subprocesses  have  been  proceduralized.  Thus,  not  only  will 
there  be  procedural  capabilities  that  depend  upon  earlier  declarative 
learning  but  there  will  also  be  declarative  learning  goals  that  cannot  be 
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realized  until  some  subprocesses  have  been  compiled. 

3.4  Complex  Performances  Require  Component  Proceduralization 

Perfetti  and  Lesgold  (1978)  applied  a  variation  of  this  notion  to  the  • 
problem  of  explaining  why  some  children  cannot  read  very  well.  Building  on 
theoretical  accounts  of  the  use  of  schemas  in  comprehension  of  discourse 
(Rumelhart  &  Ortony,  1977;  Schank  &  Abelson,  1977;  inter  alia),  they 
suggested  the  following  theoretical  view  (see  also  LaBerge  and  Samuels, 
1974):  The  reading  process  is  too  complex  to  operate  completely  at  the 
declarative  processing  level .  It  can  only  work  well  when  every  component 
that  can  be  automated  is  practiced  enough  to  be  compiled  into  an  automatic 
form. 


There  are  two  kinds  of  data  that  support  this  assertion.  First,  there 
are  numerous  studies  (e.g.,  Perfetti  &  Hogaboam,  1975;  Frederlksen,  1978) 
that  show  specific  speed-of-processing  differences  between  children  who  do 
and  those  who  do  not  read  at  normal  levels  for  their  age.  In  a  longitudinal 
study  of  reading  acquisition  (Lesgold  &  Curtis,  1981;  Lesgold  &  Resnick, 
1982),  there  are  strong  suggestions  that  overall  reading  achievement  as 
measured  by  standardized  comprehension  tests  has  accurate  and  speedy  word 
recognition  as  a  prerequisite.  The  second  kind  of  evidence  is  that  both 
text  structure  manipulations  and  global  reading  ability  differences  seem  to 
affect  learning  of  the  main  points  of  a  discourse  less  than  the  learning  of 
details.  All  of  this  is  consistent  with  a  model  of  reading  skill  in  which 
most  students  have  a  reasonable  general  plan  for  reading  but  can  only  carry 
it  out  completely  if  many  aspects  of  it  are  automated. 
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Consider,  for  example,  the  following  small  passage: 

Howard  went  to  the  bank.  He  wanted  to  buy  the  house  he 
had  seen  yesterday.  The  owner  had  said  that  the  bank 
gave  the  best  interest  rates  in  town. 

After  reading  the  first  sentence,  the  only  main  point  is  that  Howard  went  to 
the  bank.  However,  given  adequately  automated  understanding  of  bank ,  i.e., 
an  adequately  automated  bank  schema,  certain  facts  ought  to  be  activated  in 
long-term  memory,  e,.g.,  that  banks  give  mortgage  loans.  If  the  bank  schema 
is  less  automated,  the  activation  will  not  occur  till  the  second  or  third 
sentence  is  read.  If  the  recognition  capability  for  the  words  of  the  first 
sentence  are  not  automated,  short-term  memory  will  be  swamped  with,  the 
episodic  trace  of  the  word  decoding  process,  which  will  decrease  the 
effectiveness  of  automated  domain-relevant  knowledge  and  decrease  the 
likelihood  that  a  house-buying  schema  could  be  activated  enough  to  make  the 
expectation  of  going  to  the  bank  to  get  a  mortgage  salient.  Reading  between 
the  lines  (in  this  case,  not  very  far  between  the  lines)  depends  heavily  on 
the  ability  to  carry  out  very  active,  elaborated  understanding  of  what  is 
being  read.  Issues  that  have  not  yet  come  up  directly  need  to  be  expected. 
The  success  of  this  expectation  process  will  depend  on  the  extent  to  which 
components  of  the  reading  process  are  proceduralized  (automated) . 

Because  of  capacity  limitations,  reverting  to  the  declarative  level  of 
processing  may  hurt  more  than  it  helps.  Characteristically,  effective 
strategies  call  for  putting  off  decisions  as  long  as  possible  while  making  a 
mental  note  that  information  has  appeared  which  imposes  a  constraint  or  that 
may  eventually  require  a  decision.  In  almost  every  human  endeavor,  trying 
to  make  decisions  as  soon  as  new  constraints  arise  is  a  bad  policy.  It 
diverts  attention  from  systematic  planning.  This  leads  to  the  hypothesis 
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that  while  elaborated,  active  understanding  activity  is  essential  to 
adequate  comprehension  of  a  complex  text,  it  may  be  counterproductive  to  try 
to  produce  much  uncontrolled  elaboration  if  such  activity  can  only  be 
sustained  at  the  declarative  processing  level. 

3.5  Radiology 

I  turn  now  to  a  different  phenomenon  that  demonstrates  the  role  of 
practice  and  proceduralization  in  acquisition  of  skill.  Across  two 
different  empirical  investigations,  my  colleagues  and  I  have  found  a 
nonlinear,  U-shaped  relationship  among  beginning,  intermediate,  and  expert 
radiologists;  beginning  residents  and  experts  were  better  than 
intermediates.  This  does  not  happen  in  every  case,  but  it  is  a  recurrent 
result  in  our  studies. [3]  We  have  been  studying  this  phenomenon,  and  we 
think  we  understand  it,  at  least  in  part. 

'■That  follows  is  a  caricature  of  some  of  our  detailed  protocol  analyses. 
For  new  residents,  film  analysis  is  tightly  bound  to  physical  features  of 
the  film,  e.g.,  densities  of  various  textures,  sizes,  and  shapes.  These 
features  are  construed  literally.  Thus,  for  example,  a  rather  well-defined, 
dense  abnormality  of  the  sort  present  in  one  of  our  films  might  be 
interpreted  as  a  collapsed  lung,  without  any  rich  consideration  of  the 
context  in  which  it  appears  nor  of  the  medical  physiological  condition  of 
the  patient,  just  as  the  poor  reader  processes  only  the  most  central  and 
immediate  meaning  of  what  9/he  reads.  In  fact,  though,  the  actual  shadow 
cast  on  the  x-ray  film  plate  by  certain  lung  collapses  is  no  different  from 
shadows  certain  tumors  could  cast.  Only  context  can  distinguish  the  two. 
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Diagnosis  that  is  insensitive  to  context,  that  only  maps  shapes  on  the 
film  to  prototype  templates,  will  often  lead  the  diagnostician  to  error, 
because  many  alternative  structural  forms  map  to  the  same  manifestations. 
Correct  diagnosis,  when  it  occurs,  will  be  highly  dependent  on  a  fortuitous 
relationship  between  true  pathology  and  the  novices'  primary  interpretations 
of  chest  features.  But  this  can  readily  happen,  and  it  does  for  some  of  our 
cases  that  are  almost  perfect  examples . 

Intermediate  trainees  are  in  the  process  of  compiling  and  tuning  their 
ability  to  perceive  complex  anatomical  details  and  to  take  account  of 
interactions  or  constraints  imposed  by  film  context  and  by  variations  in 
film  quality.  They  are  also  developing  their  ability  to  construct  a  global 
model  of  the  patient's  medical  condition  and  the  conditions  of  film 
production.  However,  they  suffer  in  some  cases  because  their  new,  more 
complete,  schemas  assert  control  but  are  insufficiently  automated  to  finish 
the  job;  i.e.,  they  no  longer  have  the  simplistic  recognition  abilities  of 
the  new  trainee  and  they  have  not  yet  automated  the  refinements  they  have 
acquired. 

This  period  of  short-term  loss,  however,  is  a  necessary  developmental 
phase  which  ends  in  the  automated,  refined,  flexible  schemata  of  the  expert. 
Expert  interpretations  are,  once  again,  relatively  direct,  but  in  the 
process  of  their  construction,  they  come  to  incorporate  appropriate 
contextual  factors.  For  example,  the  expert  might  develop  a  set  of  "chest 
forms"  (e.g.,  the  emphysematous  chest,  the  underinflated  chest,  the  under- 
or  over-exposed  chest)  along  with  distinct  and  specially  tailored 
recognition,  interpretation  and  evaluation  rules  applicable  within  each  of 
these  forms. 
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4.0  REPRESENTATION  CONSTRUCTION 

While  much  of  what  seems  important  about  expertise  can  be  lumped  under 
the  categories  knowledge  and  practice,  there  is  one  more  aspect  that 
deserves  special  mention.  This  is  representational  skill.  Again,  the  case 
of  radiology  helps  us  make  the  point.  Radiologists  need  to  see  a  patient 
when  they  look  at  a  film,  not  just  a  complex  visual  stimulus.  All  of  their 
medical  knowledge  is  organized  around  the  human  body;  it  would  be 
counterproductive  to  reorganize  it  around  blips  on  a  film.  Also,  the 
meaning  of  any  given  film  feature  is  determined  in  part  by  surrounding 
context.  The  same  blob  will  appear  to  be  a  tumor  in  one  set  of  surrounding 
features  and  just  an  engorged  blood  vessel  in  a  different  context.  Again, 
the  best  way  to  organize  this  contextual  knowledge  is  around  principles  of 
human  physiology  and  learned  variations  (both  benign  and  pathological)  in 
human  anatomy. 

Our  analyses  of  expert  and  resident  radiologists  showed  that  subjects 
differed  in  the  precision  with  which  they  "zeroed  in"  on  the  target  feature 
within  the  film.  In  those  analyses,  we  found  that  not  all  subjects  thought 
primarily  about  a  mental  representation  of  the  patient.  Some  thought  aloud 
more  in  terms  of  film  properties. 

To  examine  this  more  systematically,  we  looked  at  all  of  the  statements 
about  locations  of  abnormalities  from  our  subjects'  discussions  of  one  of 
our  films.  These  statements  were  classified  into  four  categories  in  order 
of  increasing  anatomic  specificity:  (1)  Spatial  statements  referred  to 
two-dimensional  surface  areas  of  the  film  itself  and  were,  in  a  sense, 
non-anatomlcal .  (2)  Gross  anatomical  location  statements  referred  to 

components  of  anatomy  in  conglomerate  without  indication  of  which  specific 
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components  were  involved,  e.g.,  “hilar  vasculature."  (3)  Nominal  anatomical 
location  statements  mentioned  anatomical  components  by  name,  e.g.,  the 
"pulmonary  artery."  (4)  Finally,  target  anatomical  localization  statements 
explicitly  restricted  the  abnormality  to  an  appropriate  subpart  of  an 
anatomical  structure.  Examples  of  these  four  kinds  of  location  statements 
are  given  in  Table  1. 

Table  2  shows  the  number  of  subjects  within  each  group  who  achieved  the 
increasingly  specific  levels  of  anatomical  localization  within  their 
analyses  of  the  target  abnormality.  Spatial  location  statements  were  all 
that  some  of  our  subjects  generated.  For  example,  one  subject  spoke  of  a 
prominent  right  hllum  or  mediastinum.  Neither  term  refers  to  specific, 
systemic  anatomy.  In  contrast,  seventeen  of  nineteen  subjects  who  correctly 
detected  abnormality  used  anatomic  location  statements.  However,  these 
localizations  ranged  from  gross  anatomy  to  components  of  specific  anatomy 
within  the  target  area  of  the  film.  Table  2  shows  that  this  anatomic 
specificity  was  tied  closely  to  expertise — all  experts  mentioned  specific 
components  of  anatomy  within  the  target  region,  while  hardly  any  first  and 
second  year  residents  did  (1  of  7,  14%). 

When  these  findings  are  taken  in  conjunction  with  other  findings  in  our 
studies,  they  suggest  that  experts  rely  upon  a  mental  representation  of 
specific  anatomic  structures  in  a  specific  patient  to  separate  abnormalities 
from  other  structures  in  the  area.  Third  and  fourth  year  residents,  perhaps 
more  knowledgeable  of  the  structure  of  anatomy  within  the  target  region  than 
new  residents,  were  nonetheless  largely  unsuccessful  in  referring  x-ray 
shadows  to  this  anatomy  appropriately.  This  could  be  due  to  imprecision  in 
their  knowledge  of  the  anatomical  structures  themselves,  to  limitation  in 
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Table  1 

Examples  of  Different  Types  of  Localization  (Film  9) 


Localization  Type  Examples 

Spatial 

. . .  Prominent  right  hilum  or  mediastinum 

Anatomical:  Gross 

.  .  .  hilar  prominence  .  .  .  may  be  due  to  right  hilar  vasculature.  .  . 

Anatomical :  Nominal 

. . .  could  be  pulmonary  arterial  hypertension.  . . 

.  . .  enlarged  pulmonary  arteries  from  chronic  obstructive 
pulmonary  disease. . . 

Anatomical:  Target 

. . .  slight  density  above  right  hilum;  I  think  it's  the  azygos  vein. . 

. . .  Pulmonary  hila  themselves  not  enlarged.  .  .  fullness  in  the 
right  mediastinum. . .  a  little  above  the  hilum. . .  not  part  of 
the  aorta. . .  definitely  separate  from  the  aorta. . . 


Table  2 

Most  Detailed  Level  of  Anatomical  Localization  for  Each  Subject  on  Film  9 


Level  of 
Localization 

Residents 

1st  /  2nd  Year 

Residents 

3rd  /  4th  Year 

Experts 

Spatial 

1 

1 

0 

Anatomical 

Gross 

2 

1 

0 

Nominal 

3 

1 

0 

Target 

1 

4 

5 
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knowledge  of  how  these  structures  vary  normally  and  under  perturbation,  or, 
more  directly,  to  deficiency  in  mapping  this  anatomy  onto  radiographic 
manifestations.  First  and  second  year  residents  were,  perhaps,  more  limited 
to  recognizing  gross  visual  properties  of  the  film  and  were  equally  likely 
to  respond  to  these  with  interpretations  of  tumor  or  vascularity. 

One  of  the  most  striking  findings  from  these  data  was  that  many 
diagnostic  errors  resulted  from  a  combination  of  misperception  of  anatomy 
and  the  inability  to  see  simultaneously  two  structures  that  projected  onto 
the  same  region  of  a  film.  The  phenomenon  is  seen  clearly  in  the 
atelectasis  film  discussed  above  (Figures  1  and  2).  Recall  that  the 
triangular  (sail-shaped)  density  in  the  right  lung  is  a  critical  feature. 

It  is  also  pretty  obvious,  even  to  nonphysicians.  However,  some  of  our 
subjects  did  not  see  this  feature.  The  reason  is  that  they  attributed  part 
of  the  whitened  area  to  the  pulmonary  artery,  leaving  a  much  smaller 
abnormality,  which  they  then  thought  was  a  tumor.  This  can  be  seen  in 
Figure  3a,  which  shows  the  entire  sail-shaped  region  and  the  portion  that 
some  residents  marked  off  as  really  being  artery. 

In  outlining  critical  regions  on  the  film,  half  of  our  residents  showed 
the  pulmonary  artery  taking  up  part  of  the  sail-shaped  region.  All  then 
reported  a  smaller  abnormality  instead  of  the  sail-shaped  hallmark  of  middle 
lobe  collapse.  The  only  others  to  draw  any  pulmonary  artery  in  the 
sail-shaped  region  were  two  experts  who  correctly  drew  a  collateral  branch 
of  the  artery  that  occupied  5  to  10  percent  of  the  region,  overlapping  the 
sail  abnormality,  as  shown  in  Figure  3b.  (The  contours  in  Figure  3  are 
actual  lines  drawn  by  subjects  in  an  anatomy  sketching  task.) 
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We  can  summarize  this  result,  which  was  obtained  in  at  least  one  other 
film  as  well,  by  recasting  it  a  bit.  To  our  residents,  reading  an  x-ray 
picture  is  like  doing  an  embedded  figures  test.  At  any  instant,  each  local 
feature  in  the  film  can  only  be  assigned  to  a  single  anatomical  structure. 
For  our  experts,  though,  there  is  a  greater  ability  to  recognize  overlapping 
anatomical  structures.  Their  mental  representation  of  the  patient  can  be 
decoupled  from  film  features.  This  allows  overlapping  structures  to  be 
detected.  We  speculate  that  this  automated  capability  is  a  part  of  the 
apparently  greater  ability  of  the  expert  to  envision  the  patient's  internal 
anatomy  even  while  questioning  some  of  the  evidence  that  led  to  that 
envisionment . 

5.0  IMPLICATIONS  FOR  INSTRUCTION 

Consider  the  stereotypic  Socratic  dialogue  that  goes  on  when  a  resident 
is  questioned  during  grand  rounds.  He  or  she  proposes  a  diagnosis,  which 
turns  out  to  be  wrong.  Then  the  senior  attending  physician  asks  a  series  of 
rhetorical  questions  that  walk  the  student  through  the  correct  diagnosis. 

The  student  realizes  that  s/he  knew  the  rules  for  making  the  right  diagnosis 
but  did  not  use  them  at  the  right  time.  S/he  thinks  that  s/he  was  negligent 
or  inattentive,  but  the  kind  of  model  proposed  by  Anderson  suggests 
otherwise.  Perhaps  the  knowledge  of  the  resident  student  is  not 
sufficiently  proceduralized,  something  that  happens  only  with  practice.  Or, 
it  may  not  be  able  to  take  account  of  complex  context  dependencies  in  the 
meanings  of  manifestations  in  the  patient,  something  that  can  be  improved 
through  appropriate  variation  in  the  cases  the  resident  experiences. 
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A  more  serious  research  issue  also  arises  when  we  think  about  the 
status  of  complex  procedures  that  can  only  execute  when  many  of  their 
components  have  been  automated  (proceduralized) .  We  have  assumed  in  this 
chapter  that  expert  performance  involves  the  neatly  sequential  execution  of 
complex  procedures.  However,  the  mechanisms  of  automation  that  we  have 
discussed  apply  to  individual  productions.  That  is,  snapshots  of  small 
pieces  of  a  complex  procedure  each  become  automated.  Should  those  automated 
pieces  execute  in  a  fixed  sequence  a  number  of  times,  they  can  then  be 
composed  into  longer  sequences.  However,  there  is  a  likely  state  of  affairs 
in  the  midst  of  learning  when  the  components  of  a  procedure  are  each 
automated  but  their  sequence  is  not  yet  fully  constrained. 

In  this  state  of  affairs,  thinking  is  not  well  described  by  sequential 
procedure  descriptions.  It  is  more  like  a  Pandemonium  (Self ridge,  1959)  in 
which  many  fragments  of  the  target  activity  occur  in  parallel.  Indeed,  some 
researchers  (e.g.,  McClelland  &  Rumelhart,  1982;  Rumelhart  &  McClelland, 
1981,  1982)  have  proposed  models  for  mature  performance  which  remain 
strongly  parallel  and  fragmented.  Nonetheless,  it  seems  likely  that  linear 
plans  do  drive  problem  solving  in  experts,  at  least  to  some  extent.  An  open 
research  question  is:  Where  do  these  plans  come  from?  One  obvious  source 
is  the  composition  mechanism,  which  will  tend  to  form  recurrent,  successful 
sequences  of  processing  into  unified  procedures.  A  second  source  is 
declarative  knowledge.  Specifically,  I  suggest  that  a  verbal  plan,  a  list 
of  steps  toward  solution,  can  help  in  the  composition  process.  Such  a  plan, 
taught  only  to  the  declarative  level,  can  act  by  introducing  a  useful 
oscillation  between  self-conscious  attempts  to  achieve  successive  subgoals 
of  the  plan  and  the  Pandemonium-like  activity  of  automated  fragments.  The 
verbal  plan  components  can  act  to  entrain  the  parallel  processing  activity 
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of  automated  fragments  that  are  not  quite  reliable  enough  to  assure  an 
efficient  processing  sequence  without  such  intervention. 

If  this  is  the  case,  then  we  need  to  consider  whether  the  plans  that  we 
teach  the  student  should  be  the  same  as  a  caricature  of  the  master's 
activities  or  whether  there  are  other  alternatives  that  are  more  effective 
in  shaping  efficient  and  reliable  performance.  Glaser  (1982)  has  spoken  of 
pedagogical  theories,  theories  for  a  phenomenon  that  are  used  as  temporary 
teaching  tools  and  then  discarded  as  more  sophistication  develops.  My 
suggestion  is  a  specific  instance  of  this  idea.  Just  as  we  teach  physics 
students  Newton's  Laws  first  and  then  show  why  they  aren't  quite  adequate 
given  the  special  theory  of  relativity,  we  might  teach  simple  verbal  rules 
to  guide  problem  solving  and  then  replace  them  as  bigger  components  of  the 
overall  target  performance  are  automated. 

If  one  examines  the  training  exercises  used  by  ballet  instructors,  by 
gymnastics  coaches,  and  by  many  other  teachers,  we  see  hints  that  teachers' 
practical  wisdom  includes  this  concept.  Certain  drills  are  repeated 
regularly  even  though  they  are  not,  themselves,  target  performances. 

However,  they  may  have  the  effect  of  building  the  right  procedural 
subsequences.  Sometimes  this  involves  a  separate  activity,  like  a  ballet 
exercise;  sometimes  it  involves  a  mnemonic,  as  when  a  mother  helps  a  child 
put  on  mittens  by  saying  "thumb  in  the  thumb  place,  fingers  all  together;" 
sometimes  it  involves  verbal  coaching,  as  when  a  teacher  coaches  a  student 
through  the  proof  of  a  geometry  theorem.  In  any  case,  devising  a  theory  of 
the  principles  for  guiding  the  development  of  systematic  procedures  from 
incompletely  organized  pandemonia  of  fragmentary  productions  is  a  major  task 
for  cognitive  psychologists  who  wish  to  improve  instruction  aimed  at  high 
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levels  of  skill. 
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Footnotes 


1.  Historians  have  not  yet  verified  that  baseball  has  always  involved 
complex  cognitive  activity.  Particularly,  it  is  not  known  whether 
our  festschrif tee  engaged  in  cognitive  activity  while  pursuing 
baseball  expertise. 

2.  Truth  maintenance  refers  to  the  process  of  maintaining  consistency 
of  beliefs  as  new  assumptions  are  being  "tried  out"  (cf.  deKLleer, 
Doyle,  Steele,  and  Sussman,  1977). 

3.  All  of  our  films  involve  situations  in  which  several  alternative 
possibilities  must  be  considered.  As  John  Anderson  pointed  out  in 
comments  on  an  earlier  draft,  it  is  unlikely  that  the  learning 
curve  averaged  over  the  mix  of  films  on  which  radiologists  are 
trained  is  nonmonotone. 


The  right  lung  is  divided  into  three  connected  lobes;  the  left 
lung  has  two. 
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Figure  Captions 

Frontal  x-ray  film  of  the  chest  of  a  patient  with  a  collapsed  right 
middle  lung  lobe.  Patient  faces  forward  (his  right  is  on  your 
left). 

Illustration  of  the  location  of  the  collapsed  lung  tissue  in  Figure 

1. 

Anatomical  sketches  of  structures  seen  by  subjects  in  the 
"triangular  density"  region  of  the  film  shown  in  Figure  1.  (a) 

Residents  often  "used  up"  the  region  by  filling  it  with  pulmonary 
artery,  (b)  Experts  saw  only  a  small  collateral  of  the  pulmonary 
artery  in  the  critical  region. 


51 


Pittsburgh/Glaser  4  Lesgold  (NR  667-430  6-Dec-82 


Page 


Navy 

1  Robert  Ahlers 
Code  N711 

Human  Factors  Laboratory 
NAVTRAEQUIPCEN 
Orlando,  FL  32813 

1  CDR  Robert  J.  Biersner 
Naval  Medical  R4D  Command 
National  Naval  Medical  Center 
Bethesda,  MD  20814 


1  Dr.  Richard  Cantone 

Navy  Research  Laboratory 
Code  7510 

Washington,  DC  2C375 

1  Chief  of  Naval  Education  and  Training 
Liason  Office 

Air  Force  Human  Resource  Laboratory 
Operations  Training  Division 
WILLIAMS  AFB,  AZ  85224 

1  Dr.  Stanley  Col Iyer 
Office  of  Naval  Technology 
800  N.  Quincy  Street 
Arlington,  VA  22217 

1  CDR  Mike  Curran 
Office  of  Naval  Research 
80C  N.  Quincy  St. 

Code  270 

Arlington,  VA  22217 

1  Dr.  Carl  E.  Englund 
Naval  Health  Research  Center 
Code  8060  Environmental  Physiology  Dept 
P.0.  Box  85122 
San  Diego,  CA  92138 

1  DR.  PAT  FEDERICO 
Code  PI 3 
NPRDC 

San  Diego,  CA  92152 

1  Dr.  John  Ford 
Navy  Personnel  R4D  Center 
San  Diego,  CA  92152 


Navy 

1  Dr.  Jude  Franklin 
Code  7510 

Navy  Research  Laboratory 
Washington,  DC  20375 

1  Dr.  Mike  Gaynor 

Navy  Research  Laboratory 
Code  7510 

Washington,  DC  20375 

1  Dr.  Jim  Hollan 
Code  304 

Navy  Personnel  R  4  D  Center 
San  Diego,  CA  92152 

1  Dr.  Ed  Hutchins 

Navy  Personnel  R4D  Center 
San  Diego.  CA  92152 

1  Dr.  Norman  J.  Kerr 

Chief  of  Naval  Technical  Training 
Naval  Air  Station  Memphis  (75) 
Millington,  TN  38054 

1  Dr.  James  Lester 
0NR  Detachment 
495  Summer  Street 
Boston,  MA  02210 

1  Dr.  William  L.  Maloy 

Principal  Civilian  Advisor  for 
Education  and  Training 
Naval  Training  Command,  Code  00A 
Pensacola,  FL  32508 

1  Dr.  George  Moeller 
Director,  Behavioral  Sciences  Dept. 
Naval  Submarine  Medical  Research  Lab 
Naval  Submarine  Base 
Groton,  CT  63409 

1  Dr  William  Montague 
NPRDC  Code  13 
San  Diego,  CA  92152 

1  Naval  Ocean  R4D  Agency 
NSTL  Station 

Attn:  LCDR  J.  D.  McKendrick 
Code  335 

Bay  St.  Louis,  MO  39529 

1  Technical  Director 

Navy  Personnel  R4D  Center 
San  Diego,  CA  92152 


1  Liaison  Scientist 
Office  of  Naval  Research 
Branch  Office,  London 
Box  39 

FPC  New  York.  NY  09510 


Pittsburgh/Glaser  &  Lesgold  (NR  667-430  6-Dec-82 


Page  2 


Navy 

1  Office  of  Naval  Research 
Code  433 

800  N.  Quincy  SStreet 
Arlington,  VA  22217 

1  Office  of  Naval  Research 
Code  441NP 

800  N.  Quincy  Street 
Arlington,  VA  22217 

6  Personnel  &  Training  Research  Group 
Code  442PT 

Office  of  Naval  Research 
Arlington,  VA  22217 

1  Office  of  the  Chief  of  Naval  Operations 
Research  Development  &  Studies  Branch 
OP  115 

Washington,  DC  20350 

1  LT  Frank  C.  Petho,  MSC,  USN  (Ph.D) 

CNET  (N-432) 

NAS 

Pensacola,  FL  32508 

1  Dr.  Gary  Poock 

Operations  Research  Department 
Code  55PK 

Naval  Postgraduate  School 
Monterey,  CA  93940 

1  Dr.  Gil  Ricard 
Code  N711 
NTEC 

Orlando,  FL  32813 

1  Dr.  Worth  Scanland 
CNET  (N-5) 

NAS,  Pensacola,  FL  32508 

1  Dr.  Robert  G.  Smith 
Office  of  Chief  of  Naval  Operations 
OP-937H 

Washington,  DC  20350 

1  Dr.  Alfred  F.  Smode,  Director 
Training  Analysis  &  Evaluation  Group 
Dept,  of  the  Navy 
Orlando,  FL  32813 

1  Dr.  Richard  Sorensen 
Navy  Personnel  R4D  Center 
San  Diego,  CA  92152 


Navy 

1  Dr.  Frederick  Steinheiser 
CNO  -  0P115 
Navy  Annex 
Arlington,  VA  20370 

1  Roger  Weissinger-Baylon 
Department  of  Administrative  Sciences 
Naval  Postgraduate  School 
Monterey,  CA  93940 

1  Dr.  Douglas  Wetzel 
Code  12  ' 

Navy  Personnel  R&D  Center 
San  Diego,  CA  92152 


Pittsburgh/Glaser  &  Lesgold  (NR  667-430  6-Dec-82 


Page  3 


Marine  Corps 

1  H.  William  Greenup 
Education  Advisor  (E031) 
Education  Center,  MCDEC 
Quantico,  VA  221 34 

1  Special  Assistant  for  Marine 
Corps  Matters 
Code  100M 

Office  of  Naval  Research 
800  N.  Quincy  St. 

Arlington,  VA  22217 

1  DR.  A.L.  SLAFKOSKY 
SCIENTIFIC  ADVISOR  (CODE  RD-1 ) 
HQ,  U.S.  MARINE  CORPS 
WASHINGTON.  DC  20380 


Army 

1  Technical  Director 
l).  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Mr.  James  Baker 
Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Beatrice  J.  Farr 
U.  S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Milton  S.  Katz 
Training  Technical  Area 
U.S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Marshall  Narva 
US  Army  Research  Institute  for  the 
Behavioral  &  Social  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Harold  F.  O'Neil,  Jr. 

Director,  Training  Research  Lab 
Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Joseph  Psotka 
Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Robert  Sasmor 
U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Robert  Wisher 
Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 


Pittsburgh/Glaser  4  Lesgold  (NR  667-430  6-Dec-82 


Page  4 


Air  Force 

1  AFHRL/LRS 

Attn:  Susan  Ewing 
WPAFB 

WPAFB ,  OH  45433 

1  U.S.  Air  Force  Office  of  Scientific 
Research 

Life  Sciences  Directorate,  NL 
Bolling  Air  Force  Base 
Washington,  DC  20332 

1  Air  University  Library 
AUL/LSE  76/443 
Maxwell  AFB,  AL  36112 

1  Dr.  Earl  A.  Alluisi 
HO,  AFHRL  ( AFSC) 

Brooks  AFB,  TX  78235 

1  Bryan  Dallman 
AFHRL/LRT 

Lowry  AFB,  CO  80230 

1  Dr.  Alfred  R.  Fregly 
AFOSR/NL 

Bolling  AFB,  DC  20332 

1  Dr.  Genevieve  Haddad 
Program  Manager 
Life  Sciences  Directorate 
AFOSR 

Bolling  AFB,  DC  20332 

1  Dr.  T.  M.  Longridge 
AFHRL/OTGT 

Williams  AFB,  AZ  85224 


Department  of  Defense 

12  Defense  Technical  Information  Center 
Cameron  Station,  Bldg  5 
Alexandria,  VA  22 314 
Attn:  TC 

1  Military  Assistant  for  Training  and 
Personnel  Technology 

Office  of  the  Under  Secretary  of  Defens 
for  Research  4  Engineering 
Room  3D 129,  The  Pentagon 
Washington,  DC  20301 

1  Major  Jack  Thorpe 
DARPA 

1400  Wilson  Blvd. 

Arlington,  VA  22209 


S| 

»  ®1 
t 

»  ■*. 

1 


« 


Pittsburgh/Glaser  4  Lesgold  (NR  667-H3C  6-Dec-82 


Page  5 


Civilian  Agencies 

1  Dr.  Patricia  A.  Butler 
NIE-BRN  Bldg,  Stop  #  7 
1200  19th  St.,  NW 
Washington,  DC  20203 

1  Dr.  Susan  Chipman 

Learning  and  Development 
National  Institute  of  Education 
1200  19th  Street  NW 
Washington,  DC  20208 

1  Dr .  John  Mays 

National  Institute  of  Education 
1200  19th  Street  NW 
Washington,  DC  20208 

1  Dr .  Arthur  Melmed 
OERI 

1200  19th  Street  NW 
Washington,  DC  20208 

1  Dr.  Andrew  R.  Molnar 
Office  of  Scientific  and  Engineering 
Personnel  and  Education 
National  Science  Foundation 
Washington,  DC  20550 

1  Dr.  Judith  Orasanu 
National  Institute  of  Education 
1200  19th  St.,  N.W. 

Washington,  DC  20208 

1  Dr.  Ramsay  W.  Selden 
National  Institute  of  Education 
1200  19th  St.,  NW 
Washington,  DC  20208 

1  Chief,  Psychological  Reserch  Branch 
U.  S.  Coast  Guard  (G-P-1/2/TPH2) 
Washington,  DC  20593 

1  Dr.  Frank  Withrow 
U.  S.  Office  of  Education 
HOC  Maryland  Ave.  SW 
Washington,  DC  20202 

1  Dr.  Joseph  L.  Young,  Director 
Memory  &  Cognitive  Processes 
National  Science  Foundation 
Washington,  DC  20550 


Private  Sector 

1  Dr.  John  R.  Anderson 
Department  of  Psychology 
Carnegie-Mellon  University 
Pittsburgh,  PA  15213 

1  Dr .  John  Annett 

Department  of  Psychology 
University  of  Warwick 
Coventry  CVU  7AJ 
ENGLAND 

1  Dr.  Michael  Atwood 
Bell  Laboratories 
11900  North  Pecos  St. 

Denver,  CO  3023U 

1  Psychological  Research  Unit 
Dept,  of  Defense  (Army  Office) 
Campbell  Park  Offices 
Canberra  ACT  2600 
AUSTRALIA 

1  Dr.  Alan  Baddeley 

Medical  Research  Council 
Applied  Psychology  Unit 
15  Chaucer  Road 
Cambridge  CB2  2EF 
ENGLAND 

1  Dr.  Patricia  Baggett 
Department  of  Psychology 
University  of  Colorado 
Boulder,  CO  80309 

1  Dr.  Jonathan  Baron 
80  Glenn  Avenue 
Berwyn,  PA  19312 

1  Mr.  Avron  Barr 

Department  of  Computer  Science 
Stanford  University 
Stanford,  CA  9^305 

1  Dr.  Menucha  Birenbaun 
School  of  Education 
Tel  Aviv  University 
Tel  Aviv,  Ramat  Aviv  69978 
Israel 

1  Dr.  John  Black 
Yale  University 
Box  1 1 A ,  Yale  Station 
New  Haven,  CT  C6520 


Pittsburgh/Glaser  &  Lesgold  (MR  667-430  6-Dec-82 


Page  6 


Private  Sector 

1  Dr.  Lyle  Bourne 

Department  of  Psychology 
University  of  Colorado 
Boulder,  CO  80309 

1  Dr.  John  S.  Brown 
XEROX  Palo  Alto  Research  Center 
3333  Coyote  Road 
Palo  Alto,  CA  94304 

1  Dr .  Bruce  Buchanan 

Department  of  Computer  Science 
Stanford  University 
Stanford,  CA  94305 

1  Dr .  Robert  Buhr 
Hubbs/Seaworld  Research  Institute 
University  of  California 
1700  South  Shores  Road,  Mission  Bay 
San  Diego,  CA  92109 

1  Bundministerium  der  Verteidigjng 
-Refer at  P  II  4- 
Psychological  Service 
Postfach  1328 
D-5300  Bonn  1 
F.  R.  of  Germany 

1  Dr.  C.  Victor  Bunderson 
WICAT  Inc. 

University  Plaza,  Suite  10 
1160  So.  State  St. 

Orem,  UT  84057 

1  Dr.  Jaime  Carbonell 

Carnegie-Mellon  University 
Department  of  Psychology 
Pittsburgh,  PA  15213 

1  Dr.  Pat  Carpenter 
Department  of  Psychology 
Carnegie-Mellon  University 
Pittsburgh,  PA  15213 

1  Dr.  William  Chase 
Department  of  Psychology 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 

1  Dr.  William  Clancey 
Department  of  Computer  Science 
Stanford  University 
Stanford,  CA  94306 


Private  Sector  • 

1  Dr.  Michael  Cole 
University  of  California 
at  San  Diego 

Laboratory  of  Comparative 
Human  Cognition  -  D003A 
La  Jolla,  CA  92093 

1  Dr.  Allan  M.  Collins 
Bolt  Beranek  &  Newman,  Inc. 

50  Moulton  Street 
Cambridge,  MA  02138 

1  Dr.  Lynn  A.  Cooper 
LRDC 

University  of  Pittsburgh 
3939  O’Hara  Street 
Pittsburgh,  PA  15213 

1  Dr.  Kenneth  B.  Cross 
Anacapa  Sciences,  Inc. 

P.0.  Drawer  Q 

Santa  Barbara,  CA  93102 

1  LCOL  J.  C.  Eggenberger 
DIRECTORATE  OF  PERSONNEL  APPLIED  RESEAR 
NATIONAL  DEFENCE  HQ 
101  COLONEL  BY  DRIVE 
OTTAWA,  CANADA  K1A 

1  Dr.  Ed  Feigenbaum 
Department  of  Computer  Science 
Stanford  University 
Stanford.  CA  94305 

1  Dr.  Paul  Feltovich 

Department  of  Medical  Education 
Southern  Illinois  University 
School  of  Medicine 
P.0.  Box  3926 
Springfield,  IL  62708 

1  Professor  Reuven  Feuerstein 
HWCRI  Rehov  Karmon  6 
Bet  Hakerem 
Jerusalem 
Israel 

1  Mr.  Wallace  Feurzeig 
Department  of  Educational  Technology 
Bolt  Beranek  &  Newman 
10  Moulton  St. 

Cambridge,  MA  02238 


Pittsburgh/Glaser  &  Lesgold  (NR  667-430  6-Dec-82 


Page  7 


Private  Sector 


Private  Sector 


1  Dr.  Victor  Fields 
Dept,  of  Psychology 
Montgomery  College 
Rockville,  MD  20850 

1  Univ.  Prof.  Dr.  Gerhard  Fischer 
Liebiggasse  5/3 
A  1010  Vienna 
AUSTRIA 

1  Dr.  Dexter  Fletcher 
WICAT  Research  Institute 
1875  S.  State  St. 

Orem,  UT  22333 

1  Dr.  John  R.  Frederiksen 
Bolt  Beranek  &  Neuman 
50  Moulton  Street 
Cambridge,  MA  02138 

1  Dr.  Alinda  Friedman 
Department  of  Psychology 
University  of  Alberta 
Edmonton,  Alberta 
CAF’.DA  T6G  2E9 

1  Dr.  Michael  Genesereth 
Department  of  Computer  Science 
Stanford  University 
Stanford,  CA  94305 

1  Dr.  Don  Gentner 

Center  for  Human  Information  Processing 
University  of  California,  San  Diego 
La  Jolla,  CA  92093 

1  Dr.  Dedre  Gentner 
Bolt  Beranek  &  Newman 
10  Moulton  St. 

Cambridge,  MA  02138 

1  Dr.  Marvin  D.  Clock 
217  Stone  Hall 
Cornell  University 
Ithaca,  NY  14853 

1  Dr.  Josph  Goguen 
SRI  International 
333  Ravenswood  Avenue 
Menlo  Park,  CA  9*1025 


1  Dr.  Daniel  Gopher 
Department  of  Psychology 
University  of  Illinois 
Champaign,  IL  61820 

1  Dr.  Bert  Green 
Johns  Hopkins  University 
Department  of  Psychology 
Charles  &  34th  Street 
Baltimore,  MD  21218 

1  DR.  JAMES  G.  GREENO 
LRDC 

UNIVERSITY  OF  PITTSBURGH 
3939  O'HARA  STREET 
PITTSBURGH,  PA  15213 

1  Dr.  Barbara  Hayes-Roth 
Department  of  Computer  Science 
Stanford  University 
Stanford,  CA  95305 

1  Dr.  Frederick  Hayes-Roth 
Teknowledge 
525  University  Ave. 

Palo  Alto,  CA  94301 

1  Dr.  Kristina  Hooper 
Clark  Kerr  Hall 
University  of  California 
Santa  Cruz,  CA  95060 

1  Glenda  Greenwald,  Ed. 

Human  Intelligence  Newsletter 
P.  0.  Box  1163 
Birmingham,  MI  48012 

1  Dr.  Earl  Hunt 
Dept,  of  Psychology 
University  of  Washington 
Seattle,  WA  98105 

1  Dr.  Marcel  Just 

Department  of  Psychology 
Carnegie-Mellon  University 
Pittsburgh,  PA  15213 

1  Dr.  David  Kieras 
Department  of  Psychology 
University  of  Arizona 
Tuscon,  AZ  85721 


i 


Pittsburgh/Glaser  &  Lesgold  (NR  667-430  6-Dec-82 


Page  8 


N 


Private  Sector 

1  Dr.  Walter  Kintsch 

Department  of  Psychology 
University  of  Colorado 
Boulder,  CO  80302 

1  Dr.  Stephen  Kosslyn 
Department  of  Psychology 
Brandeis  University 
Waltham,  MA  02254 

1  Dr.  Bat  Langley 

Carnegie-Mellon  University 
Pittsburgh,  PA  15213 

1  Dr.  Marcy  Lansman 
The  L.  L.  Thurstone  Psychometric 
Laboratory 

University  of  North  Carolina 
Davie  Hall  01 3A 
Chapel  Hill,  NC  27514 

1  Dr.  Jill  Larkin 

Department  of  Psychology 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 

1  Dr.  Alan  Lesgold 
Learning  RAD  Center 
University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15260 

1  Dr.  Jim  Levin 
University  of  California 
at  San  Diego 

Laboratory  fof  Comparativ.e 
Human  Cognition  -  D003A 
La  Jolla,  CA  92093 

1  Dr.  Michael  Levine 

Department  of  Educational  Psychology 
210  Education  Bldg. 

University  of  Illinois 
Champaign,  IL  61801 

1  Dr.  Marcia  C.  Linn 
University  of  California 
Director,  Adolescent  Reasoning  Project 
Berkeley,  CA  94720 

1  Dr.  Jay  McClelland 
Department  of  Psychology 
MIT 

Cambridge,  MA  02139 


Private  Sector 

1  Dr.  James  R.  Miller 
Texas  Instruments,  Inc. 

Central  Research  Laboratory 
P.  0.  Box  226015,  MS238 
Dallas,  TX  75266 

1  Dr.  Mark  Miller 

Computer  Thought  Corporation 
1721  West  Plane  Parkway 
Plano,  TX  75075 

1  Dr.  Tom  Moran 
Xerox  PARC 
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Washington,  DC  20418 
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Bell  Laboratories 
Murray  Hill,  NJ  07974 

1  Dr.  William  B.  Rouse 
Georgia  Institute  of  Technology 
School  of  Industrial  &  Systems 
Engineering 
Atlanta,  GA  30332 

1  Dr.  David  Rumelhart 
Center  for  Human  Information  Processing 
Univ.  of  California,  San  Diego 
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New  Haven,  CT  06520 

1  Dr.  Walter  Schneider 
Psychology  Department 
603  E.  Daniel 
Champaign,  IL  61820 

1  Dr.  Alan  Schoenfeld 
Mathematics  and  Education 
The  University  of  Rochester 
Rochester,  NY  14627 

1  DR.  ROBERT  J.  SEIDEL 
INSTRUCTIONAL  TECHNOLOGY  GROUP 
HUMRRO 

?00  N.  WASHINGTON  ST. 

ALEXANDRIA,  VA  22314 

1  Mr.  Colin  Sheppard 
Applied  Psychology  Unit 
Admiralty  Marine  Technology  Est. 
Teddington,  Middlesex 
United  Kingdom 

1  Dr.  H.  Wallace  Sinaiko 
Program  Director 

Manpower  Research  and  Advisory  Services 
Smithsonian  Institution 
801  North  Pitt  Street 
Alexandria,  VA  2231 14 

1  Dr.  Edward  E.  Smith 
Bolt  Beranek  &  Newman,  Inc. 

50  Moulton  Street 
Cambridge,  MA  02138 

1  Dr.  Richard  Snow 
School  of  Education 
Stanford  University 
Stanford,  CA  94305 
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Pittsburgh/Glaser  &  Lesgold  (NR  667-430  6-Dec-82 
Private  Sector 

1  Dr.  Eliott  Soloway 
Yale  University 

Department  of  Computer  Science 

P.0.  Box  2158 

New  Haven,  CT  06520 

1  Dr.  Kathryn  T.  Spoehr 
Psychology  Department 
Brown  University 
Providence,  RI  02912 

1  Dr.  Robert  Sternberg 
Dept,  of  Psychology 
Yale  University 
Box  11A,  Yale  Station 
New  Haven,  CT  06520 

1  Dr.  Albert  Stevens 
Bolt  Beranek  &  Newman,  Inc. 

10  Moulton  St. 

Cambridge,  MA  02238 

1  David  E.  Stone,  Ph.D. 

Hazeltine  Corporation 
7680  Old  Springhouse  Road 
McLean,  VA  22102 

1  DR.  PATRICK  SUPPES 

INSTITUTE  FOR  MATHEMATICAL  STUDIES  IN 
THE  SOCIAL  SCIENCES 
STANFORD  UNIVERSITY 
STANFORD,  CA  94305 

1  Dr.  Kikumi  Tatsuoka 
Computer  Based  Education  Research  Lab 
252  Engineering  Research  Laboratory 
Urbana,  IL  61801 

1  Dr.  Maurice  Tatsuoka 
Department  of  Educational  Psychology 
University  of  Illinois 
Urbana,  IL  61801 

1  Dr.  Perry  W.  Thorndyke 
Perceptronics,  Inc. 

545  Middlefield  Road,  Suite  140 
Menlo  Park,  CA  94025 

1  Dr.  Douglas  Towne 
Univ.  of  So.  California 
Behavioral  Technology  Labs 
1845  S.  Elena  Ave. 

Redondo  Beach,  CA  90277 
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Private  Sector 

1  Dr.  Kurt  Van  Lehn 
Zerox  PARC 

3333  Coyote  Hill  Road 
Palo  Alto,  CA  94304 

1  DR.  GERSH0N  WELTMAN 
PERCEPTRONICS  INC. 

6271  VARIEL  AVE. 

WOODLAND  HILLS,  CA  91367 

1  Dr.  Keith  T.  Wescourt 
Perceptronics,  Inc. 

545  Middlefield  Road,  Suite  140 
Menlo  Park,  CA  94025 

1  William  B.  Whitten 
Bell  Laboratories 
2D-610 

Holmdel ,  NJ  07733 

1  Dr.  Christopher  Wickens 
Department  of  Psychology 
University  of  Illinois 
Champaign,  IL  61820 

1  Dr.  Mike  Williams 
Zerox  PARC 

3333  Coyote  Hill  Road 
Palo  Alto,  CA  94304 


